Statistical Comparisons of Classifiers over Multiple Data Sets

نویسنده

  • Janez Demsar
چکیده

While methods for comparing two learning algorithms on a single data set have been scrutinized for quite some time already, the issue of statistical tests for comparisons of more algorithms on multiple data sets, which is even more essential to typical machine learning studies, has been all but ignored. This article reviews the current practice and then theoretically and empirically examines several suitable tests. Based on that, we recommend a set of simple, yet safe and robust non-parametric tests for statistical comparisons of classifiers: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparison of more classifiers over multiple data sets. Results of the latter can also be neatly presented with the newly introduced CD (critical difference) diagrams.

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

On Combining Multiple Classifiers Using an Evidential Approach

Combining multiple classifiers via combining schemes or meta-learners has led to substantial improvements in many classification problems. One of the challenging tasks is to choose appropriate combining schemes and classifiers involved in an ensemble of classifiers. In this paper we propose a novel evidential approach to combining decisions given by multiple classifiers. We develop a novel evid...

متن کامل

On the use of Heronian means in a similarity classifier

This paper introduces new similarity classifiers using the Heronian mean, and the generalized Heronian mean operators. We examine the use of these operators at the aggregation step within the similarity classifier. The similarity classifier was earlier studied with other operators, in particular with an arithmetic mean, generalized mean, OWA operators, and many more. The two classifiers here ar...

متن کامل

A study of statistical techniques and performance measures for genetics-based machine learning: accuracy and interpretability

The experimental analysis on the performance of a proposed method is a crucial and necessary task to carry out in a research. This paper is focused on the statistical analysis of the results in the field of genetics-based machine Learning. It presents a study involving a set of techniques which can be used for doing a rigorous comparison among algorithms, in terms of obtaining successful classi...

متن کامل

Comparative Performances of Stochastic Competitive Evolutionary Neural Tree (SCENT) with Neural Classifiers

A stochastic competitive evolutionary neural tree (SCENT) is described and evaluated against the best neural classifiers with equivalent functionality, using a collection of data sets chosen to provide a variety of clustering scenarios. SCENT is firstly shown to produce flat classifications at least as well as the other two neural classifiers used. Moreover its variability in performance over t...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

عنوان ژورنال:
  • Journal of Machine Learning Research

دوره 7  شماره 

صفحات  -

تاریخ انتشار 2006